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ABSTRACT 

Experts' opinion is proposed as a valuable evaluation 
tool. Advantages of this method include the relative cost 
effectiveness when compared with other data collection methods. It is 
a time-saving method important in formative evaluation when a 
decision must be made concerning implementation of a course of 
action. When experts are carefully selected, their opinions add to 
the credibility of the evaluation. Exports* opinion may be used for 
the assessment of project goals and strategies which need to be 
evaluated at an early stage of the project. A systematic process for 
using experts' opinion consists of five major stages: (1) object 
identification; (2) sampling of experts; (3) presenting questions to 
experts; (4) obtaining experts' responses; and (5) synthesizing 
opinions. A large scale evaluation project, the Tel Aviv Evaluation 
(TAE) Project utilized this process. The TAE was developed to 
evaluate the proj3Cts included in the Tel Aviv Reinforcement and 
Enrichment Program to provide decision makers with relevant 
informtion to improve the Tel Aviv, Israel public schools. (DWH) 
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INTRODUCTION 

As part of the attempts to extend the scope of evaluation in recent years 
(Nevo, 1983), it has been suggested that evaluation practice not limit 
itself to the assessment of the results or outcomes of an evaluation object. 
Stake (1967) made a distinction between descriptive information and 
judgemental infprmation in evaluation. He suggested that descriptive 
information focus on intents and on observations regarding antecedents 
(prior conditions that may affect outcomes), transactions (the process of 
implementation) and outcomes of various types. Judgemental information 
in Stake 1 s approach is comprised of standards and judgements regarding 
the same antecedents, transactions and outcomes. Stuff lebeam 1 s CIPP 
Evaluation Model (Stuff lebeam, et al. f 1971; Stuff lebeam, 1974) suggests 
that evaluation locus on four kinds of information regarding each evalua- 
tion object: (a) its goals, (b) its design, (c) its process of implementa- 
tion, and (d) its outgoes. Guba and Lincoln (1981) suggested thaf the 
evaluator collect five kinds of information: (a) descriptive information 
regarding the evaluation object and its setting, (b) information respon- 
sive to concerns of relevant audiences, (c) information about relevant 
issues, (d) information on existing values, and (e) information on , 
standards relevant to the worth and merit of the evaluated object. 

In addition to the general notion that program evaluation should broaden 
its scope regarding the kinds of information to be collected on each 
evaluated- program, it has also been suggested that evaluation extend its 
perspective regarding the stage of the program in which evaluation should 
be introduced. Traditionally, the major use of evaluation has been in its 
summative - retroactive mode to serve selection and accountability. 
However, advocates of evaluation tended to point out its potential contri- 
bution to planning and management if used in a more formative - proactive 
mode, Scriven's distinction between Formative Evaluation and Summative 
Evaluation (Scriven, 1967), and Stuff lebeam 1 s distinction between proactive 



evaluation intended to serve decision making, and retroactive evaluation to 
serve accountability (Stuff lebeam, 1972) inspired evaluat;ors to introduce 
evaluation into projects at their early stages of conception and structur- 
ing. Used at the early stages of planning and implementation, evaluation 
could help clarify goals and challenge some "taken for granted' 1 assumptions 
underlying the project. It could also provide information to choose among 
alternative strategies of action or enable modification and improvement 
of current activities. 

If an evaluation of a project has to prdvidc a wide range of information, 
part of which is mainly needed at the early stages of developing the project, 
then a fast and flexible evaluation tool is required. And that's where expert 
opinion comes in. 

ADVANTAGES AND SHORTCOMINGS 

Experts 1 opinion is probably one of the oldest evaluation tools in education 
as well as other social domains. Policy-makers often seek the advice of 
experts prior to making policy decisions. Research proposals are reviewed by 
panels of experts prior to funding decisions. College and university faculty 
members are assessed by c, leagues "in their area of expertise when consi- 

O r 

dered for promotion or tenure. Manuscripts submitted for publication in 
refereed journals are reviewed by expert readers. With the prof essionaliza- 
tion of program evaluation the use of experts 1 opinion !ias also been 
recommended by many authors as a legitimate topi in systematic evaluation 
(Stake, 1967; Reinhard, 1972; Lewy, 1977; Eisener, 1979). Reinhard (1972) 
structured the use of "Advocate Teams' 1 into a systematic procedure for ,f Input 
Evaluation" according to Stuff lebeams' CIPP Evaluation Model. Eisner has 
used experts 1 opinion as a basis for the development of his Connoisseurship 
Evaluation Model (Eisener, 1979). 

The use of experts 1 opinion as an evaluation tool seems to have four major 
advantages. First, it is an inexpensive evaluation tool compared to other 



procedures of data collection such as administration of tests or questionn- 
aires, conducting observations or interviewing project participants and 
other associated persons. The cost of obtaining experts' opinion regarding 
a specific project is usually comprised of the cost of several "consult- 
ing days plus the cost of synthesizing theopinions if more than one 
expert has been used. • Second, experts 1 opinion is a. quick evaluation tool. 
In many cases timeliness is a very important consideration in conducting 
evaluation. This is especially so in the case of proactive evaluation when 
a decision ha^ to be made whether to implement a certain course of action. 
Most evaluation clients are eager to receive fast feed-back and in many 
cases it seems impossible to postpone a decision for several months or 
even y^ars until a thorough evaluation has been completed. In sur.h cases 
using experts 9 opinion might be the only feasible alternative to not doing 
an evaluation at all. Third, if experts are chosen carefully, their 
opinions might be highly appreciated by various evaluation audiences, 
thus adding to the credibility 6f the entire evaluation. Some evaluation 
clients trust experts 1 opinions much more than empirical evidence obtained' 
from a controlled experiment and analysed by means. of a sophisticated 
statistical procedure. For a novice evaluator opinions of some prestigious 
experts on the project he has been evaluating can add a lot of credibility 
to his evaluation report. The fourth advantage of experts' opinion is 
that it can be used for the evaluation of a new project without it having 
to be implemented. This feature ipakes experts' opinion an appropriate tool 
for the assessment of project goals and strategies which should be evaluated 
at an early stage to avoid the waste of resources used for implementing 
untested strategies attempting to achieve unwarranted goals. Thus, experts' 
opinion can be appropriate for Context Evaluation and Input Evaluation in 
the CIPP Model (Stuff lebeam, et al. , 1971), or for the evaluation of 
intended antecedents, intended transaction* and intended outcomes in Stake's 
Countenance Model (Stake, 1967). 
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And yet experts 1 opinions have shortcomings as well as advantages* They 
have been known for their subjectivity and inconclusiveness . The first 
shortcoming of this evaluation tool is related to the subjectivity of the 
opinions expressed by various experts affecting the reliability of the 
information provided. Another- shortcoming is related to the fact that 
experts tend sometimes to be very careful iu their responses thus hesitat- 
ing to provide conclusive answers to the qupstions presented to them. 
This might not only limit the utility of experts' opinions., but also 
make the synthesis of their opinions a very difficult t^r'*. ' thir'J 
shortcoming stems from the fact that at an early stage "in the life of 
a project (when experts 1 opinion is mostly ,uscd) it seems difficult to 
obtain a clear description of the. evaluated projec t , inc] uding explicit 
statements of its goals and strategies. The lack of a valid characteriza- 
tion of the project might make it difficult to "present 11 the project to 
experts and askSthem to review it. The last shortcoming is related to 
a possible shortage of experts in a certain area of expertise, creating 
difficulties for th^ evaluator in securing the cooperation of appropriate 
experts for his evaluation. 

Those shortcomings of experts' opinion can be minimized when used in a 
structured and. systematic way, as will be shown in the remainder of this 
article. 

A SYSTEMATIC PROCESS FOR USING EXPERTS' OPINION 

« 

The underlying assumption of this paper is tint using experts' opinion 
in a systematic way could enable us to benefit from the advantages of 
this tool while minimizing its limitations. Tn devf loping a systematic 
way for using experts' opinion five major issues have to be considered. 
They are: (a) object identif ication," (b) sampling of experts, 
(c) presenting questions to experts, (d) obtaining experts' responses, 
and Ce) synthesizing experts' opinions. These issues could also represent 
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the major stages in a systematic process of using experts' opinion. ^We 
shall describe these stages and present a large scale evaluation project 
in which these stages have been developed and applied. 



(1) Object Identification 

The object' of the evaluation (program, project, material, etc. ) 
should be sufficiently described so as Jto be* clearly identified 
when presented to experts. Such a description should include 
the goal(s) of the object, its target population, Its 
strategy, its administrative structure, its plan of action, 
its personnel and the resources at its disposition. If such 
a description is available in one of the documents of the 
evaluated object (e.g. the funding proposal), it can be used 
to present the object to the experts. If an appropriate project 
description is not avaixable, the evaluator should develop 
one. He should prepare a written description, of the projection 
the basis of his interaction with project personnel and review 
of relevant documents (if available) • He should also ask the 
project personnel to check the factual accuracy of his descrip- 
tion and make sure that it has been approved by the project 0 
director prior to its presentation to the experts. 

(2) Sampling of Experts 

A careful choice of experts is a major factor in establishing 
the credibility of their opinions. Although random sampling * 
is not feasible, in most cases an unbiased sample of experts 

can be obtained by using the following criteria: First, 

« 

experts who are or have been associated with the evaluated 
project should not be included. Experts who are known for 
their extreme support or opposition to the projsct should 
also be excluded. Second, the sample should represent a wide 
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range of expertise relevant to the nature of t a evaluated 
project* For example, the evaluation of a re - . itprehension 
individualized instruction project might rc lU experts in 
reading, in individualized instruction and in curriculum develo 
ment. Third, the .sample should include not only scholars from 
the academe, but also practitioners who practice .their 
expertise within the educational system. The credibility of 
academics and practitioners varies among different evalua- 
tion clients. Finally, if more than one expert fits each of 
the categories iu the second and third criteria and is willing 
to cooperate, random sampling should be used within each 
category. The size of the sample should be determined accord- 
ing to the size of the project and the resources available 
for the ^valuation. Usually three to ten experts are used 
for typical educational projects. 

(3) Presenting Questions to the Experts 

When the description 'of the project is presented to experts 
it should fee accompanied by a set of specific questions that 
they have to address. The use of one general non-specific 
question (.e.g. "What do you think about this project?") 
is strongly discouraged. The specific questions to be asked 
should be developed on the basis of the nature of the project 
and the information needs of the evaluation clients. The 
questiotawill usually be related to the appropriateness of 
the project goals, the worth of its strategies and plans of 
action, and the prospect of the project to affect its target 
population. The same set of questions should be used for all 
experts. However, specific sub-questions could be added to 
those presented to some or all of the experts according to 
their area of expertise. * 
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Experts have to be approached in advance to make sure th&y 
give their consent to provide opinions *on the project within 
an agreed-upon period of time. In many cases., it is easier to 
convince experts to render their services to the project if 
they commit themselves tQ the project (e.g. on the phone) 
before materials have been sent out to them. With those 
experts who fail to deliver their written responses ,on tine 
an appointment should be made to obtain their opinions orally. 
These responses have to be recorded by a member of the evalua- 
tion team. Later on the experts should be asked to check 
the factual accuracy of their recorded responses. This pro- 
cedure is intended to increase the response rate in the • 
original sample of experts, and minimize the sampling bias that 
might be created by the drop-out of experts who are "busier" 
(and also better?) than others* Experts from whom it seems 
impossible to obtain responses even orally have to be dropped 
from the sample. They should be replaced by others using 
the same sampling procedure described in stage 2. 

(5) Synthesizing Experts 1 Opinions 

For most evaluation clients it might be difficult to benefit 
from a set of opinions provided by a groi p of experts unless 
they have been synthesized into one composite summary. It is 
the responsibility of the project evaluator to conduct such a 
synthesis for the benefit of his client. Although this is a 
very tricky task requiring skills which are a combina- 

tion of craftsmanship and artistry, some guidelines can be 
suggested for the evaluator • First, the organizing framework 
for the synthesis should be the set of common evaluation 
questions that have been presented to the experts. The opinions 
should be analysed and synthesized ^y evaluation questions 
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rather than by the overall stance of the various experts (e.g. 
positive, negative, or neutral), or their area of expertise. 
After each question has been addressed in the synthesis, an 
overall conclusion regarding the entire project may also be , 
presented. Second, the evaluator should write the summary in 
his/her own words, using quotations to support his conclusions , 
and providing the complete, records of experts* opinions in an 
appendix. This would allow critical audiences to question the 
validity of the synthesis, or to obtain additional information 
according to their specific interests. Third, if feasible, thfe?"' 
synthesis should be developed independently by two members of 
the evaluation team. The differences - if any - between the two ' 
syntheses should later on be discussed in an attempt to reach 
a consensus on a single synthesis that would be presented to 
the evaluation clients. Fourth, the evaluator should remember, 
that an inconclusive answer is also^ an answer for a question 
where a conclusive answer is not warranted. Such an answer 
might also have practical implications and should therefore be 
reported without hesitaace. An inconclusive answer might suggest 
an experimental implementation of the project followed by 
evaluation, while a conclusive negative answer might suggest 
project termination, and a conclusive positive answer might be 
followed by full -implementation of the project without any 
fuVtlter evaluation. Finally, the validity of the synthesis of 
experts' opinions should v be established by confronting its con- 
clusions with those suggested by other sources of information 
such as previous evaluations of similar projects or relevant 
research studies. % Thus, the answers provided by the experts 
to at least part of the questions, should be confronted with \ 
answers to the same questions that would be inferred from a 
review of relevant research literature or from a systematic 
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meta-analysis of available research findings. Such a confronta 
tion could add to the validity of the experts' opinions as 
well as^to the validity of the synthesis of thpse opinions. 

This procesp. has been used in a large-scale evaluation project which we 
shall describe next, * k 
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USING EXPERTS' OPINION FOR THE TEL AVIV 
EVALUATION- PROJECTS AN EXPERIENCE 

i. 

V ' 

The Tel 4viv Evaluation Project (TAE) has been developed to evaluate the projects 

infcluded in the Tel Aviv Reinforcement and Enrichment ?rogram (REP) so that 

decision trtkkers in thft local, educational system coyld be provided with 

itelevan information in their attempts to improve the schools of Tel Aviv, 

In thfe 1983/84 school year, program included 12 proj'epts* three of 

which were at the kindergarten level, .five at the elementary school «level 

and four at the high school levety 

.._ • •> v- 

* . K 

Three'major evaluation questions were addressed by the TAE project: (a) are . 
the REP projects. based on acceptable educational approaches providing a 
r^sonable chance to affect the projects' target population? (b) are the 
projects beinj, implemented as planned and efficiently? and (c) do projects 
which are directly intended to change students' behaviour have an impact 
on students' achievements, their attitudes towards school and their social 
accpptance in their, classes? T^ese questions were asked both in relation to 
the whole program ad well as in relation to each single project included in 
the program. The two major tool>s~-used to address the first evaluation ques- 
tion were systematic reviews of rele vant research literature and experts' 
opinion. We shall describe here the way experts' opinion was used in the TAE , 
project to address the first -evaluation question in relation to the whole 
program, this, experience served as a basis for the development of a systematic 
process of using experts' opinion as presented in this paper * as well as a 
case study ^of its implements ticn# 
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Applying the process of using experts 1 opinion in the TAE project followed 
the stages 'presented in the previous section • 

In the first stage (object identification) we decided to use the annual 
REP plan as a description' of the project. This document has been prepared 
by the local education department of Tel Aviv, as a funding proposal for 
the city council. The proposal was an extensive document (about 60 pages) 
which included an introduction presenting the general *oalfl of the pre^ram. 
Most of it was devoted to a detailed presentation of the various rein'orce- 
ment and enrichment activities planked fox the schools of Tel Aviv, and 
the resources needed for their implementation. This was the version of the 
proposal that has been approved by tlie city council with some minor ^ 
changes In the budget. Although most of the proposal had the nature of a 
budgetary document and its introduction was not specific enough about 
the rationale of thfc project, we decided to use it as an "identification 
document 11 of the project mainly for its authenticity. We thought that 
in spite of its f^aws, it is an 'authentic representation of the jiature of 
the object of our evaluation. Therefore, this was the document that had 
been sent to the experts. 

In the second stage we- sampled our experts. We had consulted our clients 
and other stakeholders of the project "and decided that we r>eed to have in 
our sample of experts representatives in the areas of enrichment and 
reinforcement for disadvantaged students at kindergarten, elementary school 
and secondary school level. It was also apparent that only some of the 
experts should be from the academia and that the practitioners should 
represent various administrative levels in the educational system (teachers 
principals, supervisers, and high level administrators)^ In addition 
we had two constraints: persons associated with the pro. ,;ct had to he 
excluded, and the total number of experts in that sample had to be limited 
to a maximum of N ■ 10. Obviously, we had to make some combinations in 
the desired characteristics of the experts to function within the limits 



of those constraints. We ended up with a sample of nine experts who 
represented a wide range of expertise and other requested characteristics 
(even sex...) as can be seen in Figure 1. 



Figure 1 about here 



At the third stage we developed the questions for' the exports. These 
questions, presented in Figure 2 were developed on the basis of interface 
with the clients of the evaluation and our preliminary analysis of the 
project proposal. The questions represent the major concerns of the 
clients, as well as our understanding ot the evaluation problem at tTiat 
stage -f the evaluation, 

—————————— — — — — .— — - — — ... 

* Figure 2 about here 



The questions presented in Figure 2 were included in the cover letter 
that was attached to the projedt proposal that was then setnt to the 
experts • 

The experts were contacted by phone and their consent to respond to our 
request was obtained prior to the mailing of the letter and the proposal. 
They were asked to answer the questions in written form, within two weeks. 
However, only four of them responded in writing, fhe others who did not 
respond within a period of four weeks were interviewed, and their 
opinions recorded by the project staff. Those recorded responses were 
sent back to them to be checked for their accuracy, A total period of 
eight weeks was needed to obtain all the responses from the experts 
in written or oral form. 



The experts 1 opinions were synthesized according to the five questions 
that they were asked to respond to. The synthesis was submitted to the 
clients less than three months after the evaluation project had begun. 
Although it is beyond the scope of this paper to present the content of the 
responses obtained from the experts, it seems to be important to point 
out that the experts 9 opinions were perceived by the evaluation clients 
as very relevant information. The evaluation clients also seemed to 
appreciate the fact that such information has been provided less than 
three months after the beginning of the evaluation. Such speed proved 
a surprising experience for our evaluation clients f and served as a good 
start for the whole evaluation project. 

We hope that the experience reported in this paper proves valuable to 
evaluators interested in broadening the scope of their evaluations an^ 
using, experts 1 opinion in the light of its advantages and shortcomings. 
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Figure 1: The Sample of Experts 



1. 


Mr. 


M.A. 


High school principal 


2. 


Mr. 


S . B . E . 


Director of the department for disadvantaged 
students in the Ministry of Education 


3. 


Mr. 


A.G . 


Regional supervisor in the Ministry of 
Education 


4. 


Dr. 


R.M. 


Regional supervisor on kindergartens in 
the Ministry of Education 


5. 


Dr. 


D.F. 


University professor specializing in 
enrichment programs and reading 


6. 


Dr. 


Y. K. 


Educational sociologist specializing in 
culturally disadvantaged students 


7. 


Dr. 


A.S. 


Faculty member at a school of education 
specializing in the culturally disadvantated 


8. 


Dr. 


O.S. 


University president, ex dean of a school 
of education, and director of research in 
the Ministry of Education 


9. 


Ms. 




Elementary school teacher specializing 
in individualized instruction. 
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Figure 2: Questions Presented to Experts 



1. 


■ — ' \ 

Does the proposed program comply in your opinion with 
acceptable educational approaches? 


2 . 


Does REP include any activities that you would recommend 
not to implement because of a potential harm to students 
or the educational system? 


3. 


What do you think is the probability of the various projects 
included in REP to achieve their goals? Name those with 
ligh probabilities* anH those with low ones. . 

• 


4. 


To what extent does the enclosed document provide adequate 
guidance to those who have to implement the program? 


5. 


What would you suggest to change or improve in the various 
projects to increase their chances to succeed? 
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